There are always some tasks with short duration and high concurrency in the large-scale data analysis environment. How to schedule these concurrent jobs with low-latency requirement is a hot research topic. In some existing cluster resource management frameworks, the centralized schedulers cannot meet the low-latency requirement due to the bottleneck of the master node, and some distributed schedulers achieve the low-latency task scheduling, but has shortcomings in the optimal resource allocation and resource allocation conflict. By considering the needs for large-scale real-time jobs, a distributed cluster resource scheduling framework was designed and implemented to meet the low-latency requirement of large-scale data processing. Firstly, a two-stage scheduling framework and an optimized two-stage multi-path scheduling framework were proposed. Secondly, aiming at some resource conflict problems in two-stage multi-path scheduling, a task transfer mechanism based on load balancing was proposed to solve the load imbalance problems among computing nodes. At last, the task scheduling framework for large-scale clusters was simulated and verified by using actual load and a simulated scheduler. For the actual load, the scheduling delay of the proposed framework is controlled within 12% of that of the ideal scheduling. In the simulated environment, this framework has the delay of short-time tasks reduced by more than 40% compared with the centralized scheduler.

To deal with the problems of the strategies for selecting the global best position and the low local search ability, a multi-objective particle swarm optimization algorithm based on global best position adaptive selection and local search named MOPSO-GL was proposed. During the guiding particles selection in MOPSO-GL, the Sigma method and crowding distance of the particle in the archive were used and the archive member chose the guided particles in the swarm to improve the solution diversity and the swarm uniformity. Therefore, the population might get close to the true Pareto optimal solutions uniformly and quickly. Furthermore, the improved chaotic optimization strategy based on Skew Tent map was adopted, to improve the local search ability and the convergence of MOPSO-GL when the search ability of MOPSO-GL got weak. The simulation results show that MOPSO-GL has better convergence and distribution.